Results averaged over ResNet18 and ResNet50.

We find that the performance of the pre-training data varies substantially across distribution shifts, with no single data source dominating.

Our findings show interesting trends, particularly pertaining to recent methods for data curation such as synthetic data generation and lookup based on CLIP embeddings. We show that although these strategies are highly competitive for certain tasks, the curation strategy used to assemble the original ImageNet-1K dataset remains the gold standard. We anticipate that our benchmark can illuminate the path for new methods to further reduce the gap.

When deploying a machine learning model in production, it is common to encounter changes in the data distribution, such as shifts in covariates [Shimodaira, 2000], labels [Saerens et al., 2002,

Machine learning (ML) algorithms can often differ in performance across domains.

Specifically, we maintain and store a set of device-shared domain knowledge vectors, which accumulates the knowledge learned from all devices during their lifelong adaptation process.

Most machine learning algorithm designs aim to enhance label prediction accuracy.